Job Description

Synopsis

This individual will work with cross‑functional product teams to design, implement, and support modern data platforms leveraging Databricks and related technologies. The role requires expertise in Delta Lake, Unity Catalog, Delta Live Tables (DLT), and Lakeflow, combined with strong skills in PySpark, Structured Streaming, Autoloader, and DQX. A solid background in CI/CD, Terraform, DevOps, and DataOps practices is essential. The engineer will drive platform reliability, scalability, and automation across cloud environments, ensuring robust analytics and data science capabilities.

Accountabilities

• Treat the data platform as a core product, ensuring scalability, reliability, and innovation.
• Collaborate with Architecture on the development and prioritization of the Databricks‑based platform roadmap.
• Integrate new services such as Delta Lake, Unity Catalog, and DLT into the platform ecosystem.
• Prepare and maintain documentation for Databricks workflows, CI/CD pipelines, and infrastructure automation.
• Partner with stakeholders to resolve platform‑related technical issues and identify areas for improvement.
• Implement internal process improvements: automate ingestion with Autoloader, optimize streaming with Structured Streaming, and enforce data quality with DQX.
• Adopt Agile and DevOps practices, ensuring CI/CD pipelines and Terraform‑based infrastructure are production‑ready.
• Provide operational support for Databricks clusters, jobs, and pipelines.
• Perform fallout & impact analysis, ensure recovery of impacted transactions, and provide business impact reports for production incidents.
• Propose new ideas and technologies to foster continuous improvement and enhance user experience.
• Define, share, and drive adoption of best practices for DataOps and cloud‑native data engineering.

Qualifications

Education & Experience

• ​6+ years of experience as a Data/Platform Engineer, with proven expertise in Databricks and modern data lakehouse architectures.
• Graduate degree in Computer Science, Information Systems, or a related quantitative field.
• Hands‑on knowledge of PySpark, Structured Streaming, Autoloader, DQX, and Databricks workflows.
• Strong experience with CI/CD pipelines, Terraform, DevOps, and DataOps practices.
• Ability to troubleshoot and solve complex problems across distributed data systems.
• Experience working with Linux, shell scripting, public cloud platforms, and big data technologies.
• Strong communication skills and experience collaborating with cross‑functional teams in Agile environments.
• Ability to work independently with minimal supervision and eagerness to learn new technologies.